Methods and apparatus for deleting content from a storage system

ABSTRACT

One embodiment is directed to a computer system that includes primary and secondary storage systems. When a request to store a content unit is received, it may be determined whether to store the content unit on the primary storage system or the secondary storage system. In another embodiment, a computer may store information relating to an action to be performed in one of the directories of a file system that corresponds to a period of time. When the period of time to which the directory corresponds arrives, the action may be performed. In another embodiment, a content unit stored on a primary storage system may be copied to a secondary storage system in a computer system. After the content unit has been copied, the secondary storage system may send a delete request to the primary storage system to delete the content unit from the primary storage system.

FIELD OF THE INVENTION

The present invention relates to managing the storage of content.

DESCRIPTION OF THE RELATED ART

Virtually all computer application programs rely on storage that may beused to store computer code and data manipulated by the computer code. Atypical computer system includes one or more host computers that executesuch application programs and one or more storage systems that providestorage.

The host computers may access data by sending access requests to the oneor more storage systems. Some storage systems require that the accessrequests identify units of data to be accessed using logical volume andblock addresses. Such storage systems are known as “block I/O” storagesystems. Although the logical volumes presented by the storage system tothe host may not map in a one-to-one manner to physical storage devices,they are perceived by the host as corresponding to physical storagedevices, and the specification of a logical volume and block addressindicates where the referenced data is physically stored within thestorage system.

In contrast to block I/O storage systems, some storage systems receiveand process access requests that identify data or other content using acontent address, rather than an address that specifies where the data isphysically or logically stored in the storage system. As used herein, acontent address is an address that is computed, at least in part, fromthe content of its corresponding unit of content, which can be dataand/or metadata. For example, a content address for a unit of contentmay be computed by hashing the unit of content and using the resultinghash value as the content address. Storage systems that identify contentby a content address are termed content addressable storage (CAS)systems.

In a CAS system, unlike block I/O storage systems, the address by whicha unit of data is accessed (i.e., the content address) does not specifythe physical or logical location at which the unit of data is stored.Thus, in a CAS system, if the physical or logical location at which theunit of content is stored changes, the address by which host computersaccess the unit of content may remain the same. In contrast, in a blockI/O storage system, if the logical location at which the unit of contentis stored changes, host computers accessing the unit of content must bemade aware of the location change and then use the new logical locationof the unit of content for future accesses.

SUMMARY OF THE INVENTION

One embodiment is directed to a method for use in a computer systemcomprising a first storage system comprising at least one first storagedevice, a second storage system comprising at least one second storagedevice, and at least one communication link that couples the firststorage system and the second storage system, wherein the at least onesecond storage device is slower than the at least one first storagedevice. The method comprises acts of: (A) receiving, at the firststorage system, a request to store a content unit, the content unitbeing identifiable by a content address that is computed, at least inpart, from the content of the content unit; and (B) determining whetherto store the content unit on the first storage system or the secondstorage system based, at least in part, on a pre-defined policy. Anotherembodiment is directed to at least one computer readable medium encodedwith instructions that, when executed on a computer system, perform theabove-described method.

A further embodiment is directed to a computer system comprising: afirst storage system comprising at least one first storage device; asecond storage system comprising at least one second storage device thatis slower than the at least one first storage device; at least onecommunication link that couples the first storage system and the secondstorage system; and at least one controller that: receives, at the firststorage system, a request to store a content unit, the content unitbeing identifiable by a content address that is computed, at least inpart, from the content of the content unit; and determines whether tostore the content unit on the first storage system or the second storagesystem based, at least in part, on a pre-defined policy.

Another embodiment is directed to a method of scheduling, on a computer,an action to be performed on a content unit at a future time, whereinthe computer stores content units in a file system on the computer, thefile system having a plurality of directories arranged in a tree,comprising at least one root directory and a plurality of non-rootdirectories that each has a parent directory, wherein at least one ofthe plurality of directories in the tree corresponds to a period of timesubsumed by a period of time corresponding to its respective parentdirectory. The method comprises acts of: (A) storing, in at least one ofthe plurality of directories that corresponds to a period of time thatincludes the future time, information relating to the action; (B)determining, at a time related to the period of time that includes thefuture time, whether the information is stored in the at least one ofthe plurality of directories that corresponds to the period of time thatincludes the future time; and (C) when it is determined in the act (B)that the information is stored in the at least one of the plurality ofdirectories that corresponds to the period of time that includes thefuture time, performing the action. A further embodiment is directed toat least one computer readable medium encoded with instructions that,when executed on a computer, perform the above-described method.

Another embodiment is directed to a computer comprising: at least onestorage device comprising a file system having a plurality ofdirectories arranged in a tree, comprising at least one root directoryand a plurality of non-root directories that each has a parentdirectory, wherein at least one of the plurality of directories in thetree corresponds to a period of time subsumed by a period of timecorresponding to its respective parent directory; and at least onecontroller, coupled to the storage device, that stores content units inthe file system and schedules an action to be performed on a contentunit at a future time, wherein the at least one controller: stores, inat least one of the plurality of directories that corresponds to aperiod of time that includes the future time, information relating tothe action; determines, at a time related to the period of time thatincludes the future time, whether the information is stored in the atleast one of the plurality of directories that corresponds to the periodof time that includes the future time; and when it is determined thatthe information is stored in the at least one of the plurality ofdirectories that corresponds to the period of time that includes thefuture time, performs the action.

A further embodiment is directed to a method for use in a computersystem comprising a first storage system comprising at least one firststorage device, a second storage system comprising at least one secondstorage device, and at least one communication link that couples thefirst storage system and the second storage system, wherein the at leastone second storage device is slower than the at least one first storagedevice. The method comprises acts of: (A) receiving, at the secondstorage system, a first request to store a copy of a content unit storedon the first storage system; (B) in response to the first request,storing the copy of the content unit on the second storage system; and(C) sending a second request from the second storage system to the firststorage system to delete the content unit stored on the first storagesystem. Another embodiment is directed to at least one computer readablemedium encoded with instructions that, when executed on a computersystem, perform the above-described method.

Another embodiment is directed to a computer system comprising: a firststorage system comprising at least one first storage device; a secondstorage system comprising at least one second storage device that isslower than the at least one first storage device; at least onecommunication link that couples the first storage system and the secondstorage system; and at least one controller that: receives, at thesecond storage system, a first request to store a copy of a content unitstored on the first storage system; in response to the first request,stores the copy of the content unit on the second storage system; andsends a second request from the second storage system to the firststorage system to delete the content unit stored on the first storagesystem.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a system for moving content units from aprimary storage system to a secondary storage system, in accordance withone embodiment of the invention;

FIG. 2 is a flow chart illustrating an example of a process for deletingcontent units from the primary storage system;

FIG. 3 is a block diagram of a system, having one primary storage systemand multiple secondary storage systems, upon which embodiments of theinvention may be implemented;

FIG. 4 is a flow chart illustrating an example of a process by which asecondary storage system may delete content from a primary storagesystem, in accordance with one embodiment; and

FIG. 5 is a diagram of a time-based directory structure which may beused with embodiments of the invention.

DETAILED DESCRIPTION

In CAS systems, storage of large amounts of data may sometimes beexpensive. Applicants have appreciated that customers may desire lessexpensive storage. Thus, one embodiment of the invention is directed toa content addressable storage system including a primary storage systemthat provides storage on at least one more expensive storage device anda secondary storage system that provides storage on at least one lessexpensive storage device. Based upon a user-defined policy, contentoriginally stored to the primary storage system may be moved to thesecond storage system. This can be done in any of numerous ways. Forexample, the primary storage system may replicate content to thesecondary storage system, and once the content has been replicated tothe secondary storage system, the content may be automatically deletedfrom the primary storage system. When a subsequent request to access thecontent is received by the primary storage system, the primary storagesystem may determine that it no longer stores the content and the accessrequest may be serviced from the secondary storage system, which mayreceive the access request and may locate and return the content.

The servicing of read requests from the secondary storage system may beperformed in any suitable way, as the invention is not limited in thisrespect. In one embodiment, a host computer initially sends requests toaccess a content unit to the primary storage system. When the contentunit is not stored on the primary storage system, the primary storagesystem may inform the requesting host computer that the content unit isnot stored thereon. The host computer may be aware of the secondarystorage system so that when it receives the indication from the primarystorage system that the requested content unit is not stored thereon, itmay send the access request to the secondary storage system.

In another embodiment, the host computer may communicate only with theprimary storage system. When the primary storage system determines thatit does not store a requested content unit, it may forward the accessrequest (in a manner transparent to the host) to the secondary storagesystem. The secondary storage system, in response to the request, mayreturn the content unit to the primary storage system, which may thenreturn the content unit to the host computer.

Content may be replicated from the primary storage system to thesecondary storage system for any desired reason. For example, thecontent may be replicated so that in case of failure of a storage deviceor other component of the primary storage system, a backup copy of thecontent exists on the secondary storage system.

The content may also be replicated to provide a hierarchical storagemanagement (HSM) capability in the computer system. HSM is a techniqueby which data may be transparently moved from one storage medium toanother storage medium based on a user-defined policy. For example, datathat is accessed more frequently may be stored on a faster, but moreexpensive, storage device, while data that is accessed less frequentlymay be stored on a slower and less expensive storage device. That is,for example, data stored on magnetic disk drive that has not beenaccessed for a certain period of time may automatically be moved fromthe disk to tape. If a user later requests access to the data, the datamay automatically be retrieved from the tape device and transferred tothe magnetic disk drive.

For example, a primary storage system may provide a certain amount ofstorage space (e.g., disk-based), while secondary storage systemprovides additional storage space (e.g., tape-based). The amount ofstorage space offered by the primary or secondary storage systems may beequal, or the secondary can provide more or less storage space than theprimary, as the invention is not limited in this respect.

When content is replicated from the primary storage system to thesecondary storage system, the content may be deleted from the primarystorage system based on any policy and remain stored on the secondarystorage system. For example, content that is believed to be less likelyto be accessed with any frequency may be deleted from the primarystorage system to free up disk space on the primary storage system. Whenan access request is received by the primary storage system for suchcontent, the access request may be processed by secondary storage systemand the content may be retrieved therefrom.

In one embodiment, the primary storage system may be a contentaddressable storage (CAS) system. Examples of CAS systems on whichaspects of the present invention can be implemented are described in theapplications in Table 1 below. However, aspects of the invention relatedto use with a CAS system are not limited to use with any of the CASsystems described in these applications and the invention may beimplemented on any suitable CAS system.

In one embodiment, aspects of the present invention may be used with aCAS system that stores at least two different types of content units:blobs and CDFs as described in at least some of the Table 1applications. Data may be stored in logical objects referred to asblobs, while metadata (and optionally some data) is stored in logicalobjects referred to as content descriptor files (CDFs). Each blob andCDF may be identified by a content address that is computed, at least inpart, from the content of the blob or CDF. A single CDF may storemetadata for multiple blobs and may associate the metadata for a blobwith the blob, for example, by storing the content address of the blobwith the metadata. That is, the CDF may reference one or more blobs bystoring metadata for the blobs and/or the content addresses of theblobs. In one embodiment, when a CDF is moved from the primary storagesystem to the secondary storage system (e.g., by being replicated to thesecondary storage system and deleted from the primary storage system),the blobs referenced by the CDF may also be moved to the secondarystorage system. For example, when a CDF is replicated, the blobs that itreferences may be replicated and when the CDF is later deleted from theprimary storage system, all the blobs that it references may also bedeleted from the primary storage system.

Alternatively, in another embodiment, CDFs may remain on the primarystorage system, while blobs are deleted therefrom and are stored only onthe secondary storage system. Applicants have appreciated that CDFstypically use less storage space than blobs. Thus, in one embodiment,after a CDF and the blobs that it references have been replicated fromthe primary storage system to the secondary storage system, the blobsmay be deleted from the primary storage system, while the CDF remains onthe primary storage system. Thus, storage space on the primary storagesystem may be freed by deleting the logical objects that use morestorage space (i.e., the blobs). Additionally, the latency in respondingto certain access requests may be decreased or maintained by keeping theCDFs, which generally use less storage space, on the primary storagesystem. That is, if a subsequent access request for a CDF is received,the CDF need not be retrieved from the secondary storage system becauseit has not been deleted from the primary storage system.

Content may be replicated from the primary storage system to thesecondary storage system in any suitable way, as the invention is notlimited in this respect. For example, in one embodiment, the primarystorage system may include replication software that replicates logicalobjects (e.g., blobs and CDFs), also referred to as content units, tothe secondary storage system. The replication software may maintain alist of logical objects stored on the primary storage system that havenot yet been replicated to the secondary storage system and mayreplicate each logical object on the list to the secondary storagesystem. This may be done in any suitable way. For example, thereplication software may send a write request for a logical object fromthe primary storage system to the secondary storage system to replicatethe logical object to the secondary storage system. After a logicalobject is replicated, it may be removed from the list. When new logicalobjects are received by the primary storage system for storage, they maybe added to the list of logical objects not yet replicated.

Once content has been replicated from the primary storage system to thesecondary storage system, the content may be deleted from the primarystorage system at any suitable time and in any suitable way, as theinvention is not limited in this respect. In one embodiment, a deletepolicy engine may be used to perform this task. An example of a computersystem 100 including a primary storage system 101, a secondary storagesystem 103, and a delete policy engine 105 is shown in FIG. 1. Primarystorage system 101 may replicate content units to secondary storagesystem 103 in any of the ways discussed above. Delete policy engine 105may be configured to delete content units from primary storage system101 based on any desired policy, as the invention is not limited in thisrespect. In one embodiment, delete policy engine 105 may be configuredto delete content units based on a temporal characteristic (e.g., anycontent units that were stored on primary storage system more than threemonths ago).

Delete policy engine 105 may delete content units in any suitable way,as the invention is not limited in this respect. FIG. 2 is a flow chartof an example of one process that may be used to delete content unitsfrom primary storage system 101. However, the process of FIG. 2 is onlyone example of a process that may be used and the invention is notlimited to using the illustrative process of FIG. 2.

The process of FIG. 2 begins at act 201, where delete policy engine 105sends a time-based query request to primary storage system 101 toidentify content units stored on primary storage system 101 before acertain period of time. A query request is a request to identify contentunits that meet one or more criteria specified in the request and atime-based query request is one where the criterion specified in therequest is a time period during which content units were stored on thestorage system. Thus, a time-based query request is a request that thestorage system identify content units stored on the storage systemduring the time period specified in the request. As mentioned above,delete policy engine may be configured to delete content units stored onthe storage system before a certain time. Thus, the time-based queryrequest may request that primary storage system 101 identify contentunits stored before this time.

In one embodiment, the delete policy engine may use a different timeperiod in the time-based query request for different virtual pools onprimary storage system 101. Virtual pools are discussed in detail inapplication Ser. Nos. 10/910,985; 10/911,330; 10/911,248; 10/911,247;and 10/911,360, listed in Table 1 below. A virtual pool is a logicalgrouping of content units. Content units may be logically groupedtogether in virtual pools for any suitable reason. For example, contentunits may be grouped together to control access to certain contentunits. That is, a first host computer may be permitted only to accesscontent units in Virtual Pool A, while a second host computer may bepermitted only to access content units in Virtual Pool B. Thus, forexample, delete policy engine may send a time-based query request thatrequests identifiers of all content units in Virtual Pool A storedduring a first time period (e.g., before three months ago), but send atime-based query request that requests identifiers of all content unitsin Virtual Pool B stored during a different time period (e.g., beforeone month ago). Thus, content units stored in Virtual Pool A that are atleast three months old may be deleted, while content units in VirtualPool B that are at least one month old may be deleted.

After the time-based query request is sent, the process then continuesto act 203, where delete policy engine receives the identifiers fromprimary storage system 101 that identify content units stored on primarystorage system 101 before the time specified in the request. Deletepolicy engine 105 may add these identifiers to a list of identifiers tobe deleted. In one embodiment in which primary storage system 101 is aCAS system, the identifiers may be content addresses for content unitsthat were stored on the storage system before the specified time.

The process next continues to act 205, where delete policy engine 105verifies that each content unit identified by an identifier in the listis stored on secondary storage system 103. This verification may beperformed in any suitable way, as the invention is not limited in thisrespect. For example, in one embodiment for use with a secondary storagesystem that supports an “exists” request, delete policy engine 105 maysend an exists request that includes the identifier for a content unitto secondary storage system 103. An exists request is a request for astorage system to verify that a content unit identified by an identifierspecified in the request is stored on the storage system. In oneembodiment, delete policy engine may send a separate exists request tosecondary storage system for each identifier in the list. Alternatively,the delete policy engine may include multiple identifiers in one or moreexists requests, such that secondary storage system 103 verifies thatthe content unit corresponding to each identifier in the request isstored thereon.

When delete policy engine verifies that each content unit identified byan identifier in the list is stored on secondary storage system, theprocess continues to act 207, wherein delete policy engine 105 may senda delete request for each of the content units to primary storage system101, which may delete the content units. In one embodiment, primarystorage system may allow a reason for deletion to be specified in adelete request. This may be done for any suitable reason. For example,primary storage system may maintain an audit log of deletions thatrecords information about each deleted content unit, so that if acontent unit is deleted (i.e., from both primary storage system 101 andsecondary storage system 103) and it is later desired to access thedeleted content unit, the time at which the content unit was deletedand/or the reason for deletion may be determined. Thus, in accordancewith one embodiment, when delete policy engine deletes a content unitfrom primary storage system 101, delete policy engine 101 may specify areason for deletion that may later be used to deny the delete from beingreplicated to secondary storage system 101, as discussed below in moredetail. For example, delete policy engine may specify “DPE” as thereason for deletion, which may indicate to the secondary storage systemthat the content unit identified in a delete request should not bedeleted.

The audit log may be implemented in any suitable way, as the inventionis not limited in this respect. In one embodiment, the audit log may beimplemented using a reflection for each deleted content unit. Forexample, when the storage system receives a request to delete a contentunit, the storage system may create another content unit, referred toherein as a reflection, to replace the content unit and includeinformation that may be used to track the deletion of the content unit.The reflection may be stored on the storage system (or elsewhere) andthe content unit requested for deletion may be deleted or designated fordeletion at a later time. The reflection may include any suitableinformation. For example, the reflection may include a portion (or all)of the content of the content unit to be deleted, informationidentifying the content unit to be deleted, and/or audit information.The audit information may include, for example, the time of deletion ofthe content unit, the user or application program that generated therequest to delete the content unit, the network address of the computerthat generated the request to delete the content unit, the size of thecontent unit to be deleted, and/or any other suitable information.

In one embodiment, the reflection may also include a flag that indicatesthat the reflection is a reflection and/or not another type of contentunit (e.g., a blob or CDF). The flag may be included in the reflectionin any suitable way, as the invention is not limited in this respect.For example, the flag may be part of the content of the reflection. Inembodiments of the invention where the reflection is stored on thestorage system as a file in a file system, the flag may be part of thefile name of the reflection. However, the embodiment directed to storingthe reflection is not limited to including the flag in the file name ofa file storing a reflection, nor to even storing a reflection in a filesystem file. In embodiments of the invention implemented on a CASsystem, the reflection may be assigned a content address computed, atleast in part, from the content of the reflection. When used with suchembodiments, the flag indicating that the content unit is a reflectionmay form a part of the content address of the reflection or may beprovided in any other manner. The CAS system may store content units asfiles in a file system and use the content address of the content unitas the file name (or part of the file name) of the file for the contentunit. When used with such a system, the flag indicating that a contentunit is a reflection may be located in the content address of thereflection and/or the file name of the file for the reflection.

In one embodiment, the reflection may identify the content unit that itcorresponds to so that it can be determined to which content unit theinformation stored in the reflection pertains, and so that theinformation stored in the reflection can be found. A reflection of aparticular content unit may identify the content unit in any suitableway. For example, the content of the reflection may include informationidentifying the content unit (e.g., the file name of the content unitwhen it is stored in a file system). In embodiments of the invention foruse with a CAS system, the content address of the content unit may beincluded in the reflection. For example, the content address for thedeleted content unit may be included in the content of the reflection ormay be included in the content address of the reflection. Reflectionsare described in greater detail in application Ser. Nos. 11/034,613,11/034,737, and 11/034,732, listed in Table 1 below.

As discussed in greater detail below, reflections may be used toindicate that a content unit that is stored on both the primary andsecondary storage systems and is subsequently deleted from the primarystorage system should not be deleted from the secondary storage system.Reflections are only an example of one way in which this may beaccomplished. The invention is not limited to using reflections toprevent replication of delete requests from the primary storage systemto the secondary storage system, as this may be done in any suitableway.

Content may be distributed between the primary and secondary storagesystems in any suitable way, as the invention is not limited in thisrespect. For example, in one embodiment, all content may initially bewritten to the primary storage system and subsequently copied and/ormoved to the secondary storage system. Alternatively, content mayinitially be written to either the primary or secondary storage system.

In one embodiment, secondary storage system 103 is configured as areplication target for primary storage system 101. That is, contentunits that are stored on primary storage system 101 may be replicated tosecondary storage system 103. After being replicated, content units maybe deleted from primary storage system for any desired reason. Forexample, as described above, a content unit that is still in use andthat has been replicated to secondary storage system 103 may be deletedto free up disk space on primary storage system 101, with it beingdesirable to retain a copy of the content unit on secondary storagesystem 103 to enable future access. Alternatively, a content unit thatis no longer in use and is no longer needed may be deleted entirely,such that it is desirable to also delete the content unit from thereplication target (i.e., secondary storage system 103). Thus, in oneembodiment, delete requests received by primary storage system are alsoreplicated to its replication target (i.e., secondary storage system103). For example, when a request to delete a content unit is receivedby primary storage system 101, primary storage system 101 may delete thecontent unit and then replicate the delete request to secondary storagesystem 103 to delete the content unit therefrom.

Deletes may be replicated from primary storage system 101 to secondarystorage system 103 in any suitable way, as the invention is not limitedin this respect. For example, when a delete request is received byprimary storage system 101, primary storage system 101 may store deleteaudit information for the content unit (e.g., time of deletion, reasonfor deletion, and/or any other suitable information) and may delete thecontent unit. The primary storage system 101 may then send a deleterequest for the content unit to secondary storage system 103 and mayinclude the delete audit information along with the delete request.

In one embodiment, the reason for deletion specified in the deleterequest may be used to prevent secondary storage system 103 fromdeleting content units which are still in use and which, though intendedfor deletion from primary storage system 101, are not intended to bedeleted from secondary storage system 103. For example in theillustrative process of FIG. 2, at act 207, the delete policy engine maysend a delete request to delete a content unit to primary storage system101, specifying “DPE” as the reason for deletion (i.e., that the deleteis one initiated by the delete policy engine and not by a host and/orapplication program). The “DPE” reason may be used as a signal tosecondary storage system 103 to not delete the content unit. Primarystorage system 101 may delete the content unit and forward the deleterequest to its replication target (i.e., secondary storage system 103).Secondary storage system 103 may receive the delete request andrecognize that the reason for the deletion is “DPE.” Because the reasonfor deletion is “DPE,” secondary storage system 103 may not delete thecontent unit.

In another embodiment, when a delete request with “DPE” specified as thereason for deletion is received by the primary storage system 101, theprimary storage may recognize that the reason for the deletion is “DPE”and may not propagate the delete request to the secondary storagesystem.

In the example above, the delete policy engine is used in a computersystem where primary storage system 101 replicates delete requests tosecondary storage system 103. It should be appreciated that theinvention is not limited in this respect, as the secondary storagesystem need not be configured as a replication target for primarystorage system and the invention is not limited to any particularimplementation.

In the examples above, the allocation of content units between theprimary and secondary storage systems is managed by replicating writesand deletes from the primary storage system to the secondary storagesystem and informing the secondary storage system to ignore certainreplicated deletes. However, the invention is not limited to a system inwhich writes and/or deletes are replicated as content units may beallocated between the primary and secondary storage system in anysuitable way. In embodiments in which deletes are not replicated fromthe primary storage system to the secondary storage system, it may notbe necessary to specify a reason for deletion that indicates to thesecondary storage system to not process replication of the deleterequest.

As discussed above, at act 205 (FIG. 2) delete policy engine 105 mayverify that content units to be deleted are replicated to secondarystorage system 103 before sending delete requests to primary storagesystem 101 to delete the content units from primary storage system 101.If one or more content units to be deleted are not yet stored onsecondary storage system 103, delete requests for these content unitsmay not be sent to primary storage system 101. In one embodiment, deletepolicy engine may note that these content units are not yet replicatedto secondary storage system 103 and may subsequently check periodicallyto determine if the content units have been replicated. Once a contentunit has been replicated, delete policy engine may send a request toprimary storage system 101 to delete the content unit. Alternatively,delete policy engine 105 may simply not send requests to delete thesecontent units from primary storage system 101 until another time-basedquery request is issued. When delete policy engine 105 issues anothertime-based query request, primary storage system 101 may again identifythese content units and delete policy engine may again attempt to verifythat the content units are stored on secondary storage system 103.

In one embodiment, delete policy engine may be configured to sendtime-based query requests at a particular interval (e.g., once per week)to determine if there are any content units stored on primary storagesystem 101 that are to be deleted. However, the invention is not limitedin this respect, as the delete policy engine may send time-based queryrequests at any suitable frequency and in any suitable manner.

In the example above, the phrase “DPE” was used as a reason for deletionto signal to secondary storage system 103 that a delete request for oneor more content units should be ignored. It should be appreciated thatany suitable information may be used to connote that the delete policyengine is the reason for the deletion, as the invention is not limitedin this respect.

The delete policy engine 105 may be implemented in any suitable way, asthe invention is not limited in this respect. In one embodiment, deletepolicy engine may be software code (e.g., a program, microcode, etc.).The delete policy engine software code may be executed on any suitablecomputer (as indicated by the dotted lines of delete policy engine 105in FIG. 1), as the invention is not limited in this respect. Forexample, the delete policy engine may be executed on primary storagesystem 101, secondary storage system 103, a computer external to bothprimary storage system 101 and secondary storage system 103, and/orelsewhere. In embodiments in which delete policy engine software code isexecuted on a computer external to both primary storage system 101 andsecondary storage system 103, the computer may be, for example, anappliance in computer system 100 or may be a computer external tocomputer system 100.

In another embodiment, rather than using a separate entity (e.g., deletepolicy engine 105) to delete content units from the primary storagesystem, the primary storage system may itself be aware of the HSMpolicy. For example, the primary storage system may be configured tooperate according to a particular HSM policy. For example, a policy maybe specified that indicates that content units stored more than onemonth prior should be copied to the secondary storage system and deletedfrom the primary storage system. The primary storage system mayperiodically determine which content units stored thereon are more thana month old, copy these content units to the secondary storage system,and delete these content units from the primary storage system. Becausethe primary storage system is aware that these deletions are for HSMpurposes and not because the content units are no longer needed, primarystorage system may not replicate (e.g., forward) the delete requests tothe secondary storage system.

In one embodiment, a single HSM policy may be specified for all of thecontent units stored on the storage system. However, the invention isnot limited in this respect, as more than one HSM policy may bespecified for the storage system, with each HSM policy relating to aparticular set of content units. For example, in embodiments in whichthe primary storage system has multiple virtual pools, different HSMpolicies may be provided for different virtual pools.

In one embodiment, the computer system may have more than one secondarystorage system. For example, as shown in FIG. 3, computer system 300includes a primary storage system 301, a first secondary storage system303, and a second secondary storage system 305. Content units moved fromprimary storage system 301 (e.g., for HSM purposes) may be stored oneither secondary storage system 303, secondary storage system 305, orboth storage system 303 and storage system 305. Distribution of contentunits among the primary and secondary storage systems can be done in anysuitable way, including those discussed above. For example, primarystorage system 301 may periodically determine which content units storedthereon are more than a month old (or any other suitable time periodspecified in the HSM policy), copy these content units to eithersecondary storage system 303, secondary storage system 305 or both, andmay delete these content units from the primary storage system 301. Asin the example above, because primary storage system 301 is aware thatthese deletions are for HSM purposes and not because the content unitsare no longer needed, primary storage system may not replicate (e.g.,forward) the delete requests to the secondary storage system(s).

When a host computer sends an access request to primary storage system301 for a content unit that has been moved to secondary storage system303 and/or secondary storage system 305, primary storage system 301 maydetermine that it does not store the requested content unit and theaccess request may be presented to secondary storage system 303 and/orsecondary storage system 305 for processing.

The access request sent by the host computer to primary storage system301 may be any type of access request, as the invention is not limitedin this respect. For example, the access request may be a read request,a write request, a query request, or any other type of access request.

The presentation of the access request that fails in the primary storagesystem to the secondary storage system may be performed in any suitableway, as the invention is not limited in this respect. For example, inone embodiment, when primary storage system 301 determines that it doesnot store the requested content unit, it may indicate to the hostcomputer that the requested content unit is not stored thereon. The hostcomputer may then send the access request to secondary storage system303. If secondary storage system 303 does not store the content unit,host computer may then send the access request to secondary storagesystem 305. In another embodiment, when primary storage system 301indicates to the host computer that it does not store the requestedcontent unit, the host computer may send the access request concurrentlyto both secondary storage system 303 and secondary storage system 305.The above technique is one wherein the host has visibility to thesecondary and primary storage systems. In some systems, it may bedesirable to have the host only interact with the storage system 300itself, rather than the computers therein.

Thus, in another embodiment, storage system 300 may perform the accessrequest fail over. This can be done in numerous ways and the inventionis not limited in this respect. In one embodiment, when primary storagesystem 301 determines that it does not store the requested content unit,it may forward the access request to one or both of the secondarystorage systems. The access request may either be sent first tosecondary storage system 303 and then, if secondary storage system 303does not store the content unit, to secondary storage system 305, or maybe sent concurrently to both secondary storage system 303 and secondarystorage system 305. The secondary storage system that stores the contentunit may, in response to the request, return the content unit to primarystorage system 301, which may then return the content unit to the hostcomputer.

In another embodiment, when the storage system 300 copies a content unitto secondary storage system 303 and/or secondary storage system 305 anddeletes the content unit from primary storage system 301, the storagesystem 300 may create a placeholder for the deleted content unit on theprimary storage system 301. The placeholder may indicate to whichsecondary storage system(s) the content unit has been copied. Thus, whena host computer requests access to the content unit, the primary storagesystem 301 may find the placeholder for the content unit, determine thatthe content unit is deleted, and return the placeholder (or a portion ofthe placeholder) to the host computer. The host computer may use theinformation in the placeholder to determine which secondary storagesystem stores the content unit and may send the access request to thesecondary storage system indicated in the placeholder.

The placeholder may implemented in any suitable way, as the invention isnot limited in this respect. For example, in one embodiment, theplaceholder may be a file. In embodiments in which primary storagesystem 301 is a CAS system, the file name may include all or a part ofthe content address of the deleted content unit to which it corresponds.

As discussed above, the access request received by the primary storagesystem may be a query request, which is a request to identify contentunits that satisfy one or more criteria specified in the request. Aquery request may be received and processed in any suitable way, as theinvention is not limited in this respect. In one embodiment, a queryrequest is received and processed by both the primary storage system andthe secondary storage system. The primary storage system and thesecondary storage system may each return the content units that satisfythe criterion or criteria specified in the request. As discussed above,copies of content units stored on the primary storage system may also bestored on the secondary storage system (e.g., when a content unit isreplicated from the primary storage system to the secondary storagesystem, but has not yet been deleted from the primary storage system).When a content unit stored on both the primary storage system and thesecondary storage system satisfies the criterion or criteria specifiedin the query request, the content unit may be identified twice inresponse to the query request. In one embodiment, duplicate entries(i.e., content units identified by both the primary storage system andsecondary storage system) may be filtered out so that the applicationprogram that initiated the request does not receive duplicates inresponse to a query request.

A query request may be sent to both the primary and secondary storagesystems in any suitable way, as the invention is not limited in thisrespect. In one embodiment, the host computer may send the request toboth the primary and secondary storage systems. In another embodiment,the host computer may send the request to the primary storage systemwhich may forward the request to the secondary storage system.

The response to a query request may be returned to the host in anysuitable way, as the invention is not limited in this respect. In oneembodiment, the primary and secondary storage systems may each send aresponse to the query request to the host computer and duplicates may befiltered by software on the host computer (e.g., an applicationprogramming interface that is used by the application program tocommunicate with the storage systems) before the results are returned tothe host computer. In another embodiment, the secondary storage systemmay send its response to the query request to the primary storagesystem, which may combine its response with the response received fromthe secondary storage system, remove duplicates, and return the combinedresponse to the host. Alternatively, the primary storage system may sendits response to the secondary storage system, which may combine theprimary storage system's response with its response, remove duplicates,and return the combined response to the host.

In the example of FIG. 3, computer system 300 includes two secondarystorage systems (i.e., secondary storage system 303 and secondarystorage system 305). However, it should be appreciated that theinvention is not limited in this respect, as computer system 300 mayhave one, three, or any other suitable number of secondary storagesystems.

In another embodiment of the invention, deletion of content units movedfrom a primary storage system to a secondary storage system may beperformed by the secondary storage system. This may be done in anysuitable way, as the invention is not limited in this respect. Forexample, when a content unit is replicated to the secondary storagesystem, the secondary storage system may send a delete request to theprimary storage system to delete the content. The delete request may besent immediately or the secondary storage system may schedule a deleteevent for a future time and send the delete request when the future timearrives.

An example of a process by which the secondary storage system may deletea content unit from the primary storage system is shown in FIG. 4. Theprocess of FIG. 4 begins at act 401, where the secondary storage systemreceives a replication request from the primary storage system to storea content unit. The process then continues to act 403, where thesecondary storage system stores the content unit. The process nextcontinues to act 405, where the secondary storage system schedules adeletion event for a future time. The future time may be, for example, atime defined by an HSM policy as a time at which the content unit shouldbe removed from the primary storage system. Use of this technique may bedesirable, for example, when it is desired to create a copy of a contentunit on the secondary storage system shortly after it is stored on theprimary storage system (e.g., for back-up purposes), but it is alsodesired to keep a copy on the primary storage system for some duration(e.g., a time period when the content unit is most likely to beaccessed).

The process next continues to act 407, wherein the process waits untilthe future time at which the delete event is scheduled has arrived. Whenthe future time has arrived, the process continues to act 409, where thesecondary storage system sends a delete request for the content unit tothe primary storage system. After act 409, the process ends.

The process shown in FIG. 4 is an example of a process that may be usedfor deleting one content unit from the primary storage system. However,the invention is not limited to using a single process for a singlecontent, as the deletion, by the secondary storage system, of contentunits stored on the primary storage system may be implemented in anysuitable way. For example, a first process may be used to schedulefuture delete events for content units, while a second processperiodically checks a queue of these delete events and sends deleterequests to the primary storage system when the scheduled time for adelete event arrives.

Secondary storage system may schedule the delete request in any suitableway, as the invention is not limited in this respect. In one embodiment,secondary storage system may store content units in a time-baseddirectory. A time based directory structure is one in which directoriesand files are organized based, at least in part, on the time at whichthe content units were stored and/or created by the source (e.g., anapplication program). Any suitable time based directory structure may beused, as the invention is not limited in this respect. In oneembodiment, the time-based directory structure may be organized as aplurality of hierarchical directories, where each directory represents aperiod of time and each subdirectory represents a period of time that issubsumed by the period of time of its parent directory. Files may bestored in the bottom-level or “leaf” directories (i.e., directories thatrepresent the smallest units of time) and the leaf directory in which aparticular file is stored may be selected based on a time of storage ora time of creation of the file.

An example of such a time based directory structure is shown in FIG. 5.FIG. 5 shows a directory structure 500 having six levels (labeledL1-L6), wherein directories at level L1 designate the year in which thefile was stored, directories at level L2 designate the month,directories at level L3 designate the day, directories at level L4designate the hour, directories at level L5 designate the minute, anddirectories at level L6 designate the second. It should be appreciatedthat the entire hierarchy of directories in directory structure 500 is,for the sake of clarity, not expanded and that each non-leaf directoryshown in FIG. 5 may have one or more subdirectories that are not shownin FIG. 5.

As discussed above, files may be stored in the leaf directory thatcorresponds to the time at which they were stored and/or created. Thus,for example, a file stored on Jan. 1, 2005 at 11:00 PM may be stored inthe L6 directory 501, which has a path of /2005/January/01/23/00/00,wherein “2005” is the L1 directory corresponding to the year 2005,“January” is a subdirectory of the 2005 directory corresponding to themonth of January 2005, the subdirectory “01” of the “January” directoryis the L3 directory corresponding to the 1^(st) day of January 2005, thesubdirectory “23” of the “01” directory is the L4 directorycorresponding to the 24^(th) hour of the 1^(st) day of January 2005, thesubdirectory “00” of the “23” directory is the L5 directorycorresponding to the 1^(st) minute of the 24^(th) hour of the 1^(st) dayof January 2005, and the subdirectory “00” of the “00” L5 directory isthe L6 directory corresponding to the 1^(st) second of the 1^(st) minuteof the 24^(th) hour of the 1^(st) day of January 2005.

A content unit may be stored in the time-based directory thatcorresponds to the time at which the content unit was received forstorage by the secondary storage system. Examples of systems whichemploy a time-based directory structure are described in greater detailin application Ser. Nos. 11/107,520, 11/107,063, and 11/107,194, listedbelow in Table 1, each of which is incorporated by reference herein inits entirety.

When a content unit is replicated to a secondary storage system thecontent unit may be stored in the time-based directory of directorystructure 500 that corresponds to its time of receipt by the secondarystorage system. Additionally, a “To-Be-Deleted” file may be placed inone of the time-based directories that corresponds to a future time. Thefuture time may be a time when it is desired to delete the content unitfrom the primary storage system and may be specified by a user-definedHSM policy on the secondary storage system. The storage system may havea single HSM policy for all content units stored thereon or may have aseparate HSM policy for each virtual pool on the storage system, asdiscussed above.

The “To-Be-Deleted” file may specify the content unit to be deleted fromthe primary storage system (e.g., using the content address of thecontent unit or otherwise). The secondary storage system mayperiodically scan directories in the directory structure that correspondto the current time for “To-Be-Deleted” files. When a TBD file is found,the secondary storage system may send a request to the primary storagesystem to delete the content unit identified by the “To-Be-Deleted”file. For example, a To-Be-Deleted (TBD) file 503 may be stored indirectory 501, which corresponds to Jan. 1, 2005 at 11:00 PM. The TBDfile 503 may be stored in directory 501 at a time prior to Jan. 1, 2005at 11:00 PM and may indicate that a request to the primary storagesystem to delete the content unit identified in the TBD file should besent on or after Jan. 1, 2005 at 11:00 PM.

In the example of FIG. 5, the TBD file 503 is stored in a leafdirectory. However, the invention is not limited in this respect, as TBDfiles may be stored in any suitable directory in the time-baseddirectory structure. In one embodiment, time-based directories may havenon-time-based subdirectories, referred to as TBD subdirectories. TheTBD files may be stored in the TBD subdirectories. The scheduled time ofdeletion for a content unit identified in a TBD file stored in a TBDsubdirectory may be the time corresponding to the parent directory ofthe TBD subdirectory.

In the examples above, a time-based directory is used to aid in thescheduling of delete events by storing TBD files in directoriescorresponding to a planned future time of deletion. However, theinvention is not limited to scheduling future events in this manner, asthe delete events may be scheduled in any suitable way.

In some of the examples above a content address was used as theidentifier in identifying content units. The invention is not limited touse with a content address as any suitable identifier may be used.

The above-described embodiments of the present invention can beimplemented on any suitable computer or system. Examples of suitablecomputers and/or systems are described in the patent applications listedbelow in Table 1 (collectively “the CAS applications”), each of which isincorporated herein by reference. It should be appreciated that thecomputers and systems described in these applications are only examplesof computers and systems on which the embodiments of the presentinvention may be implemented, as the invention is not limited toimplementation on any of these content addressable storage systems, orto content addressable storage systems at all.

TABLE 1 Title Serial No. Filing Date Content Addressable 09/236,366 Jan.21, 1999 Information, Encapsulation, Representation, And Transfer AccessTo Content 09/235,146 Jan. 21, 1999 Addressable Data Over A NetworkSystem And Method For 09/391,360 Sep. 7, 1999 Secure Storage TransferAnd Retrieval Of Content Addressable Information Method And ApparatusFor 10/731,790 Dec. 9, 2003 Data Retention In A Storage System MethodsAnd Apparatus 10/731,613 Dec. 9, 2003 For Facilitating Access To ContentIn A Data Storage System Methods And Apparatus 10/731,796 Dec. 9, 2003For Caching A Location Index In A Data Storage System Methods AndApparatus 10/731,603 Dec. 9, 2003 For Parsing A Content Address ToFacilitate Selection Of A Physical Storage Location In A Data StorageSystem Methods And Apparatus 10/731,845 Dec. 9, 2003 For Generating AContent Address To Indicate Data Units Written To A Storage SystemProximate In Time Methods And Apparatus 10/762,044 Jan. 21, 2004 ForModifying A Retention Period For Data In A Storage System Methods AndApparatus 10/761,826 Jan. 21, 2004 For Extending A Retention Period ForData In A Storage System Methods And Apparatus 10/762,036 Jan. 21, 2004For Indirectly Identifying A Retention Period For Data In A StorageSystem Methods And Apparatus 10/762,043 Jan. 21, 2004 For IndirectlyIdentifying A Retention Period For Data In A Storage System Methods AndApparatus 10/787,337 Feb. 26, 2004 For Increasing Data Storage CapacityMethods And Apparatus 10/787,670 Feb. 26, 2004 For Storing Data In AStorage Environment Methods And Apparatus 10/910,985 Aug. 4, 2004 ForSegregating A Content Addressable Computer System Methods And Apparatus10/911,330 Aug. 4, 2004 For Accessing Content In A Virtual Pool On AContent Addressable Storage System Methods and Apparatus For 10/911,248Aug. 4, 2004 Including Storage System Capability Information In AnAccess Request To A Content Addressable Storage System Methods AndApparatus 10/911,247 Aug. 4, 2004 For Tracking Content Storage In AContent Addressable Storage System Methods and Apparatus For 10/911,360Aug. 4, 2004 Storing Information Identifying A Source Of A Content UnitStored On A Content Addressable System Software System For 11/021,892Dec. 23, 2004 Providing Storage System Functionality Software System For11/022,022 Dec. 23, 2004 Providing Content Addressable Storage SystemFunctionality Methods And Apparatus 11/022,077 Dec. 23, 2004 ForProviding Data Retention Capability Via A Network Attached StorageDevice Methods And Apparatus 11/021,756 Dec. 23, 2004 For ManagingStorage In A Computer System Methods And Apparatus 11/021,012 Dec. 23,2004 For Processing Access Requests In A Computer System Methods AndApparatus 11/021,378 Dec. 23, 2004 For Accessing Information In AHierarchical File System Methods And Apparatus 11/034,613 Jan. 12, 2005For Storing A Reflection On A Storage System Method And Apparatus For11/034,737 Jan. 12, 2005 Modifying A Retention Period Methods AndApparatus 11/034,732 Jan. 12, 2005 For Managing Deletion of Data MethodsAnd Apparatus 11/107,520 Apr. 15, 2005 For Managing The Storage OfContent Methods And Apparatus 11/107,063 Apr. 15, 2005 For Retrieval OfContent Units In A Time-Based Directory Structure Methods And Apparatus11/107,194 Apr. 15, 2005 For Managing The Replication Of Content MethodsAnd Apparatus 11/165104 Jun. 23, 2005 For Managing the Storage OfContent In A File System Methods And Apparatus 11/165103 Jun. 23, 2005For Accessing Content Stored In A File System Methods And Apparatus11/165102 Jun. 23, 2005 For Storing Content In A File System

The above-described embodiments of the present invention can beimplemented in any of numerous ways. For example, the embodiments may beimplemented using hardware, software or a combination thereof. Whenimplemented in software, the software code can be executed on anysuitable processor or collection of processors, whether provided in asingle computer or distributed among multiple computers. It should beappreciated that any component or collection of components that performthe functions described above can be generically considered as one ormore controllers that control the above-discussed functions. The one ormore controllers can be implemented in numerous ways, such as withdedicated hardware, or with general purpose hardware (e.g., one or moreprocessors) that is programmed using microcode or software to performthe functions recited above.

In this respect, it should be appreciated that one implementation of theembodiments of the present invention comprises at least onecomputer-readable medium (e.g., a computer memory, a floppy disk, acompact disk, a tape, etc.) encoded with a computer program (i.e., aplurality of instructions), which, when executed on a processor,performs the above-discussed functions of the embodiments of the presentinvention. The computer-readable medium can be transportable such thatthe program stored thereon can be loaded onto any computer environmentresource to implement the aspects of the present invention discussedherein. In addition, it should be appreciated that the reference to acomputer program which, when executed, performs the above-discussedfunctions, is not limited to an application program running on a hostcomputer. Rather, the term computer program is used herein in a genericsense to reference any type of computer code (e.g., software ormicrocode) that can be employed to program a processor to implement theabove-discussed aspects of the present invention.

It should be appreciated that in accordance with several embodiments ofthe present invention wherein processes are implemented in a computerreadable medium, the computer implemented processes may, during thecourse of their execution, receive input manually (e.g., from a user).

The phraseology and terminology used herein is for the purpose ofdescription and should not be regarded as limiting. The use of“including,” “comprising,” “having,” “containing”, “involving”, andvariations thereof, is meant to encompass the items listed thereafterand additional items.

Having described several embodiments of the invention in detail, variousmodifications and improvements will readily occur to those skilled inthe art. Such modifications and improvements are intended to be withinthe spirit and scope of the invention. Accordingly, the foregoingdescription is by way of example only, and is not intended as limiting.The invention is limited only as defined by the following claims and theequivalents thereto.

1. In a computer system comprising a first storage system comprising atleast one first storage device, a second storage system comprising atleast one second storage device, and at least one communication linkthat couples the first storage system and the second storage system,wherein the at least one second storage device is slower than the atleast one first storage device, wherein the second storage system storescontent units in a file system on the second storage system, the filesystem having a plurality of directories arranged in a tree, comprisingat least one root directory and a plurality of non-root directories thateach has a parent directory, wherein at least one of the plurality ofdirectories in the tree corresponds to a period of time subsumed by aperiod of time corresponding to its respective parent directory, amethod comprising acts of: (A) scheduling, at the second storage system,a time of deletion for a content unit from the first storage system bystoring, in at least one of the plurality of directories thatcorresponds to a period of time that includes the scheduled time ofdeletion, an indication to delete the content unit from the firststorage system; (B) receiving, at the second storage system, a firstrequest to store a copy of the content unit stored on the first storagesystem; (C) in response to the first request, storing the copy of thecontent unit on the second storage system; and (D) sending a secondrequest from the second storage system to the first storage system todelete the content unit stored on the first storage system.
 2. Themethod of claim 1, wherein the act (D) further comprises acts of:determining, at a time related to the period of time that includes theschedule time of deletion, whether the indication is stored in the atleast one of the plurality of directories that corresponds to the periodof time that includes the scheduled time of deletion; and when it isdetermined that the indication is stored in the at least one of theplurality of directories that corresponds to the period of time thatincludes the scheduled time of deletion, sending the second request. 3.At least one computer readable medium encoded with instructions that,when executed on a computer system, performs a method, the computersystem comprising a first storage system comprising at least one firststorage device, a second storage system comprising at least one secondstorage device, and at least one communication link that couples thefirst storage system and the second storage system, wherein the at leastone second storage device is slower than the at least one first storagedevice, wherein the second storage system stores content units in a filesystem on the second storage system, the file system having a pluralityof directories arranged in a tree, comprising at least one rootdirectory and a plurality of non-root directories that each has a parentdirectory, wherein at least one of the plurality of directories in thetree corresponds to a period of time subsumed by a period of timecorresponding to its respective parent directory, the method comprisingacts of: (A) scheduling, at the second storage system, a time ofdeletion for a content unit from the first storage system by storing, inat least one of the plurality of directories that corresponds to aperiod of time that includes the scheduled time of deletion, anindication to delete the content unit from the first storage system; (B)receiving, at the second storage system, a first request to store a copyof the content unit stored on the first storage system; (C) in responseto the first request, storing the copy of the content unit on the secondstorage system; and (D) sending a second request from the second storagesystem to the first storage system to delete the content unit stored onthe first storage system.
 4. The at least one computer readable mediumof claim 3, wherein the act (D) further comprises acts of: determining,at a time related to the period of time that includes the schedule timeof deletion, whether the indication is stored in the at least one of theplurality of directories that corresponds to the period of time thatincludes the scheduled time of deletion; and when it is determined thatthe indication is stored in the at least one of the plurality ofdirectories that corresponds to the period of time that includes thescheduled time of deletion, sending the second request.
 5. A computersystem comprising: a first storage system comprising at least one firststorage device; a second storage system comprising at least one secondstorage device that is slower than the at least one first storagedevice, wherein the second storage system comprises a file system inwhich the second storage system stores content units, the file systemhaving a plurality of directories arranged in a tree, comprising atleast one root directory and a plurality of non-root directories thateach has a parent directory, wherein at least one of the plurality ofdirectories in the tree corresponds to a period of time subsumed by aperiod of time corresponding to its respective parent directory; atleast one communication link that couples the first storage system andthe second storage system; and at least one controller that: schedules,at the second storage system, a time of deletion for the content unit;stores, in at least one of the plurality of directories that correspondsto a period of time that includes the scheduled time of deletion, anindication to delete the content unit from the first storage system;receives, at the second storage system, a first request to store a copyof a content unit stored on the first storage system; in response to thefirst request, stores the copy of the content unit on the second storagesystem; and sends a second request from the second storage system to thefirst storage system to delete the content unit stored on the firststorage system.
 6. The computer system of claim 5, wherein the at leastone controller: determines, at a time related to the period of time thatincludes the schedule time of deletion, whether the indication is storedin the at least one of the plurality of directories that corresponds tothe period of time that includes the scheduled time of deletion; andwhen it is determined that the indication is stored in the at least oneof the plurality of directories that corresponds to the period of timethat includes the scheduled time of deletion, sends the second request.